Pre-trained Language Models

Abstract This chapter presents the main architecture types of attention-based language models, which describe distribution tokens in texts: Autoencoders similar to BERT receive an input text and produce a contextual embedding for each token. Autoregressive models GPT subsequence as input. They token predict next In this way, all can successively be generated. Transformer Encoder-Decoders have task translate sequence another sequence, e.g. translation. First they generate by autoencoder. Then these embeddings are used autoregressive model, sequentially generates output tokens. These usually pre-trained on large general training set often fine-tuned specific task. Therefore, collectively called Pre-trained Language Models (PLM). When number parameters gets large, instructed prompts Foundation Models. further sections we described details optimization regularization methods training. Finally, analyze uncertainty model predictions how may explained.